Open-architecture file system

ABSTRACT

An open-architecture transactional file system is described. In one described implementation, a file system comprises a plurality of interchangeable and/or customizable logic layers configured to manage operations on behalf of an application when the application makes data requests to a data source. Functionality associated with the file system is augmentable by substituting and/or modifying one or more of the plurality of interchangeable and/or customizable logic layers, respectively, enabling the file system to function with many different types of data sources.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present patent application is a Continuation-In-Part of U.S. patentapplication Ser. No. 10/087,672, entitled “Open Architecture FlashDriver,” by Aasheim, et al., having a filing date of Feb. 27, 2002 nowU.S. Pat. No. 7,533,214, and is commonly assigned herewith (hereinafterreferred to as “Aasheim '672”). The contents of the aforementionedapplication are fully incorporated herein by reference.

TECHNICAL FIELD

This invention relates to file systems, and more particularly, to anopen-architecture transactional file system.

BACKGROUND

A file system is typically used by an operating system or programapplications to locate, name, organize and store files. “Files” arenamed collections of information in many data formats, such as aprogram, data used by a program application, user-created documents,etc. The file system, itself, consists of files, directories, andinformation needed to locate and access files. Most file systems includefunctionality needed to translate requests for files (such as readingand writing data) from application programs into low-level, sectororiented tasks that are understood by a driver and used forcommunicating with an underlying storage media, such as a hard disk,floppy diskette, Zip drive, flash drive, and so forth.

Most current file systems tend to use rigid architectures that behavethe same regardless of the underlying storage medium. Accordingly, tofacilitate communication between most file systems and different typesof storage media, it is usually necessary to create specific driversadapted to the particular type or brand of underlying media. Typically,the drivers enable a file system to interface with a particular mediumby providing a device-specific control program to communicate with themedium in response to commands received from the file system.

A drawback of current file system architectures is that they tend toincorporate methodologies that were developed decades ago. The filesystems were originally intended for use with storage mediums thatoperate in a stable operating environment, such as mainframe computersand stationary personal computers. However, when used in conjunctionwith less traditional computer devices, such as thin clients (e.g.,personal digital assistants (PDAs), tablet PCs, etc.), wirelesshandsets, set-top boxes and other non-traditional computer devices,current file systems tend to operate inefficiently. For example, many ofthe latest computer gadgets tend to be battery-operated and frequentlyexperience catastrophic system failures such as a power-failure event.Unfortunately, current file systems tend to use boot sequencingtechniques that were originally intended for use with media found inmore traditional computing environments, such as a workstation orpersonal computer communicating with a hard disk. These boot sequencingtechniques tend to be very slow and require an inordinate amount of timeto recover from a catastrophic event. With emerging computer devices,where power failures and network disconnections are common, such bootsequencing techniques take too much time and, are therefore,undesirable.

Another disadvantage of current file systems is that they tend to bemedia specific. That is, to adapt a file system to function with aparticular brand or type of media is often necessary to design (orredesign) one or more drivers to enable the file system to interfacewith the storage medium. As a result, some original equipmentmanufacturers (OEMs) of computer devices deploy vender supplied driversassociated with the vendor supplied storage medium. This results inhaving to develop intermediary drivers that are type specific tointerface between the file system and the vendor supplied driver(s). Ifan OEM desires to change from one vendor's supplied storage media to adifferent vendor's supplied storage media, then the OEM typically has todevelop new intermediary type specific drivers to interface with the newvendor supplied driver(s). This process is labor intensive, subject toerrors and delays, and is expensive. As a result, many OEMs avoid havingto change the particular brand or type of storage medium and becometied-in to a particular type/brand of storage medium.

Another drawback associated with current file systems is that they oftenuse storage techniques inherently detrimental to operatingcharacteristics of the storage medium. For example, repetitively storingto a specific area of the medium may damage and prematurely destroy thelongevity of certain media. Many traditional file systems tend to usetables and other techniques that are located in fixed locations of thestorage medium, which can be extremely harmful to certain types of datasources such as flash memory.

The aforementioned examples are only a few of the many limitationsassociated with current file systems. Accordingly, current file systemtechniques are unable to address an ever increasing demand for morerobust and adaptable file systems.

SUMMARY

An open-architecture transactional file system is described. In onedescribed implementation, a file system comprises a plurality ofinterchangeable and/or customizable logic layers configured to manageoperations on behalf of an application when the application makes datarequests to a data source. Functionality associated with the file systemis augmentable by substituting and/or modifying one or more of theplurality of interchangeable and/or customizable logic layers,respectively, enabling the file system to function with many differenttypes of data sources.

BRIEF DESCRIPTION OF THE DRAWINGS

The detailed description is described with reference to the accompanyingfigures. In the figures, the left-most digit(s) of a reference numberidentifies the figure in which the reference number first appears.

FIG. 1 shows a computer in which an open-architecture transactional filesystem may be implemented.

FIG. 2 shows a more detailed representation of the file system shown inFIG. 1.

FIG. 3 shows an open-architecture transaction file system that includesthree interchangeable and/or customizable logic layers.

FIGS. 4 and 5 illustrate logical representations of example NAND and NORflash memory, respectively.

FIG. 6 shows a transactional file system implemented for a flash memorymedia such as those described in FIGS. 4 and 5.

FIG. 7 shows modules configured for the media abstraction layer of thetransactional file system.

FIG. 8 shows how a meta-information storage module stores data on theflash medium.

FIG. 9 illustrates a method for a storing metadata in-line with data.

FIG. 10A shows an exemplary implementation of a data structure generatedby the file system.

FIG. 10B shows a data structure which is the same as the data structureshown in FIG. 10A, except its contents have been updated.

FIG. 11 illustrates a method used to track data on a flash memory mediumwhen the file system performs write requests.

FIG. 12 shows a flash memory medium viewed and/or treated as acontinuous circle by the file system.

FIG. 13 illustrates another view of a flash memory medium viewed as acontinuous circle.

FIG. 14 illustrates a method used by a sector manager to determine thenext available free sector location for the file system to store data onthe flash medium.

FIG. 15 illustrates another view of a flash memory medium viewed as acontinuous circle.

FIG. 16 shows how the file system stores meta-information on a flashmedium when a computer shuts-down (e.g., powers-off or reboots)according to a normal shutdown mode to safeguard against a power failureevent.

FIG. 17 is a method for detecting a power failure event.

FIG. 18 shows how the file system stores meta-information on the flashmedium to safeguard against partial data being written to a physicalsector of the medium due to a power failure event.

FIG. 19 is a method for ensuring data integrity in the event of apower-failure event.

FIG. 20 shows transaction information stored at an arbitrary location ona flash medium.

FIG. 21 shows a method for implementing a transaction log and theability to recover transactions in the event of a power-failure eventthrough transaction playback.

FIG. 22 illustrates a dynamic look-up data structure used to track datastored in the flash memory medium.

FIG. 23 illustrates a method for dynamically allocating look-up datastructures for tracking data on the flash memory medium.

DETAILED DESCRIPTION

Introduction

To overcome the inefficiencies and problems described in the Backgroundsection above, the following description introduces the broad concept ofa new open-architecture transactional file system that supports almostany type of data source. A “data source” as used herein means any typeof object (either physical or virtual) that possesses the ability toread and write data. Examples of data sources include, but are notlimited to: flash memory, hard disks, removable media, networks,Ethernet, and other related sources of data. As indicated, the datasource does not have to be a storage media, but can includecommunication data sources as well. The file system uses interchangeableand/or customizable logic layers that can be substituted, removed, ormodified to augment the behavior of the file system itself. Usinginterchangeable and/or customizable logic layers permits the file systemto support many different operating characteristics exhibited by themany different types of data sources.

The following description also introduces the broad concept of storingdata in a particular memory location and also storing metadata in thesame memory location with the data, where the metadata is used aspointer to link the data with additional data in another memorylocation. This avoids the pitfalls of storing an allocation table in afixed region of flash memory medium among other problems.

The following description also introduces the broad concepts of usingmeta-information in locations of a flash memory medium for detectingpower failure events, storing transaction logs, providing transactionplayback after a power failure event, ensuring data integrity after apower failure, and storing data (such as tables) spanning severalphysical locations.

The following description also introduces the broad concept of using adynamic amount of memory to store logical-to-physical sector addressmappings. Thus, the amount of memory needed to track data stored in theflash memory medium can be minimized. Additionally, through the use ofcounters associated with the dynamic amount of memory, it is possible tovery quickly determine (without having to search through an entire datastructure) whether to allocate more memory for storinglogical-to-physical sector address mappings.

File System Architecture

FIG. 1 shows a computer 100 in which an open-architecture transactionalfile system 104 (referred to generally as “file system 104”) may beimplemented. The computer 100 represents various different generalpurpose or special purpose computing system configurations, includingbut not limited to personal computers, server computers, hand-held orlaptop devices, portable communication devices, tablet PCs,televisions/set-top boxes, wireless devices, printers, photocopiers,multiprocessor systems, microprocessor systems, microprocessor-basedsystems, programmable consumer electronics, gaming systems, multimediasystems, the combination of any of the above example devices, and othersmart devices.

Computer 100 includes at least one processor 102 and memory 106.Computer 100 may also include a local data source 108(1) and/or may alsobe connected to a remote data source 108(2) via a communication link110, such as a network, wireless link, or some type of othercommunication link. Computer 100 may include more than one data sourceand/or be connected to more than one data source. A data source,referred to generally as reference number 108, is an object (eitherphysical or virtual) that possesses the ability to read and write data.Examples of data sources include, but are not limited to: flash memory,hard disks, removable media, networks, Ethernet, and other relatedsources of data.

Other elements such as power supplies, keyboards, touch pads, I/Ointerfaces, displays, LEDs, audio generators, vibrating devices, and soforth are not shown in FIG. 1, but could easily be a part of theexemplary computer 100. Additionally, although not shown, a system busor point-to-point connections, typically connects the various componentswithin computer 100.

Memory 106 may include volatile memory (e.g., RAM) and/or non-volatilememory (e.g., ROM, PCMCIA cards, etc.). In some implementations, memory106 is used as part of computer's cache, permitting application data tobe accessed quickly without having to permanently store data in anon-volatile memory device.

An operating system 112 is resident in the memory 106 and executes onprocessor 102. An example operating system implementation includes theWindows®CE operating system from Microsoft® Corporation, but otheroperation systems can be selected from one of many operating systems,such as DOS, UNIX, etc. For purposes of illustration, programs and otherexecutable program components such as the operating system areillustrated herein as discrete blocks, although it is recognized thatsuch programs and components reside at various times in differentstorage components of the computer including data source(s) 108, and areexecuted by processor of the computer 100.

One or more application programs 114 are loaded into memory 106 and runon operating system 112 or on other devices in communication withcomputer 100. Examples of application programs include, but are notlimited to, email programs, word processing programs, spreadsheetsprograms, Internet browser programs, Web services and so forth.

Although shown in memory 106, file system 104 can reside in one or moreapplications, as part of the operating system or independently asseparate code on a computer-readable medium that executes in conjunctionwith a virtual object, hardware, and/or firmware device. In theexemplary implementation, file system 104 resides as a component in theform of computer-executable instructions and/or logic within operatingsystem 112, that when executed serves as a logical interface modulebetween other application programs 114 and data source 108.

File system 104 is generally responsible for performing transactions onbehalf of operating system 112 or one or more application programs 114.When data source 108 is in the form of a storage device, such as a harddrive or flash memory device, file system 104 is responsible forstoring, retrieving, organizing files, and performing other relatedtasks.

File system 104 may also be responsible for communicating with a datasource 108 that is not a traditional storage medium per se, such as anetwork or other computer via a network, in accordance with a protocolestablished by either the file system 104 or the data source 108. In anyevent, file system 104 is capable of accessing, sending, and/or storinginformation at a logical level in accordance with the conventions of theoperating system and/or one or more application programs 114, but mayalso be configured to meet specific low level communication demands ofthe data source 108, with or without the use of a driver.

Accordingly, file system 104 is implemented to function as a directinterface between the computer 100 (via operating system 112 and/orapplication program 114) and any type of data source 108. Because filesystem 104 supports any type of data source 108 through its uniquearchitecture to be described, file system 104 is considered mediaagnostic and can easily be ported for use with many different types ofdata sources 108. As shall be described in more detail, file system 104enables power-failure safe data operations and can be configured toperform tasks particular to the type and/or brand of data source. Forexample, file system 104 (i) maintains data integrity, performswear-leveling (if the data source is some type of the flash medium),(ii) minimizes data loss during a power interruption to computer 100 orloss of connection with a data source 108, and (iii) permits OEMs ofcomputers 100 to support their respective data sources regardless of themanufacturer. Ultimately, file system 104 has the ability to read,write, erase, and manage files in conjunction with any type of datasource.

FIG. 2 shows a more detailed representation of file system 104 shown inFIG. 1. File system 104 includes logic layers 202(1), . . . , 202(N).Each of the logic layers, referred to generally as reference number 202,represent layers of abstraction containing various operating parametersper layer. For instance, the topmost layer 202(1), being closest to theoperating system 112, application programs 114, or computer 100, servesas an entry point to file system 200. Whereas, the bottom most layer202(N), being closest to data source 108, represents the layer thatdirectly accesses the data source 108.

When a data request is received at the entry point layer 202(1) to filesystem 200, each of the logical layers may be responsible for performingdifferent duties (i.e., functions, tasks, etc.) in response to the datarequest. As used herein, a “data request” generally means reading datafrom the data source and/or writing data to the data source. Typically,the upper level logic layers 203 perform higher level managerialoperations and the lower level logic layers 205 are assigned moredetailed specific operations relating to communicating with the datasource 108. In some instances, the upper logic layers 203 may operate inconjunction with lower level logic layers 205 by passing tasksassociated with servicing a data request down the chain of layers (e.g.,from 202(1) down to 202(2) down to 202(3), and so forth) to the lowerlevel logic layers 205. In other instances the upper logic layers 203may operate independently and service the data request directly withoutpassing the request down to the lower level layers 205 to handle. Stillin other instances, the upper logic layers 203 may generate their ownrequests in response to the initial data request, and pass them down tolower level logic layers 205 to handle. Thus, the logic layers 202 as awhole manage operations on behalf of an application 116, or operatingsystem 112, when either the operating system or application(s) makes adata request to data source 108.

Each of logic layers 202 is also interchangeable. As used herein,“interchangeable” means that each logic layer can be removed altogetheror another logic layer having different operating characteristics can besubstituted in its place. Accordingly, functionality (i.e., operationalbehavior) of the file system 104 is augmentable by substituting orremoving one or more of the logic layers 202.

For instance, by having well defined interfaces between each logiclayer, it is possible to have different types of the same level logicallayer that can be selected from a set of possible layers to plug intothe file system 104 and change the operational behavior of file system104. For example, it may be desirable to substitute logic layer 202(1)with a logic layer 202(M), when the data source is changed from one typeof medium to another. Accordingly, logic layer 202(1) can be removedfrom file system 104 and logic layer 202(M) can be plugged-in, in placeof logic layer 202(1). Thus, having interchangeable logic layers permitsthe file system to support many different operating characteristicsexhibited by the many different types of data sources that may besupported by file system 104.

Generally, each logic layer 202 can be replaced as long as it satisfiesthe interface(s) for the layer above it (if there is a layer above it)and below it (if there is a layer below it). For example, logic layer202(N) for interfacing with one type of data source can be replaced witha different logic layer that supports a different data source. Moreover,a set of layers, such as 202(2) through 202(5) for example, can also bereplaced, combined, or removed so long as the upper most layer in theset (e.g., 202(2)), satisfies the interface for the layer above it inthe file system 200.

Each of logic layers 202 is also customizable. As used herein,“customizable” means that each logic layer can be programmed (e.g.,configured) to behave in accordance with different operatingcharacteristics. In other words, one or more of the logic layers 202 areprogrammably configurable by a user (such as a manufacturer of computer100 or developer of a data source 108). A set of programmable entrypoints 204, permits the user to program one of more of the logic layersto initialize or augment the operational behavior of file system 104. Inone implementation, an I/O module 206 contains specific code for theuser to select in order to customize one or more of the logic layers202. For example, if the user desires to customize the file system 104to function in accordance with a particular algorithm selected by theuser to communicate with a particular data source, the user through theset of programmable entry points 204 via the I/O module 206, mayconfigure the lowest layer logic layer 202(N) (or other layers) todirectly interface with the data source according to the particularalgorithm. Thus, having customizable logic layers permits the filesystem to support many different operating characteristics exhibited bythe many different types of data sources that may be supported by filesystem 104.

FIG. 3 shows an open-architecture transaction file system 300(hereinafter referred to as “file system 300”) that includes threeinterchangeable and/or customizable logic layers. In particular, filesystem 300 includes an entry logic layer 304, a media abstraction layer306, and a media interface layer 308. Although the exemplaryimplementation is shown to include these three elements, variousfeatures from each them may be selected to carry out some of the morespecific implementations described below. So while the describedimplementation shows three distinct layers of logic, many of thetechniques described below can be implemented without necessarilyrequiring all or a portion of the features from either layer of logic.Furthermore, the techniques may be implemented without having the exactdivision of responsibilities as described below.

Entry logic layer 304 serves as the entry point to file system 300 fromprogram application(s), the operating system, and/or computer. Likelogic layer 202(1) shown in FIG. 1, entry logic layer 304 is the uppermost layer of file system 300. In the exemplary implementation, entrylogic layer 304 includes a set of Application Protocol Interfaces (APIs)that are accessible by program application(s), the operating system,and/or computer. In other words, the APIs are contained as code withinentry logic layer 304 and are exposed by file system 300 for programapplications, the operating system, and the computer to use. Examples ofthe types of operations that may be supported by layer 304 include: (i)creating a file, (ii) deleting a file, (iii) writing to a file, and (iv)other file operations typically performed by a program application, etc.Thus, entry logic layer 304 is responsible for implementing the codenecessary to perform requested operations on behalf of a callingapplication wherein the calling application includes not only programapplications, but the operating system as well as the computer.

Connecting the entry logic layer 304 to other layers of file system 300is a dispatch interface 305, which is a component of the entry logiclayer 304, but can also be implemented as a separate logic layer orwithin other layers. The purpose of the dispatch interface 305 is toforward all media specific API data requests to the media abstractionlayer 306. The media abstraction layer 306 is then responsible forensuring that all data requests are carried out by it or by one or moreother layers. Dispatch interface 305 in one exemplary implementationincludes code in the form of APIs. Dispatch interface 305 handlestransactions as well as standard data requests in the form of readingand writing data to the data source 310. Dispatch interface 305 alsoincludes code to maintain a file system directory structure and servicestorage and retrieval requests for files, etc.

Media abstraction layer 306 is configured to manage input and outputdata requests received from entry logic layer 304 via dispatch interface305. Media abstraction layer 306 serves as a translation layer to filesystem 300 and is responsible for many different functions. In oneimplementation, media abstraction layer 306 is responsible for (i)handling all input/output (I/O) requests from entry logic layer 304,(ii) performing catastrophic event detection (such as detecting a powerfailure or a network disconnection) and correction, (iii) maintainingdata integrity in the event of a catastrophic event, (iv) performinglogical to physical transformations, and (v) maintaining a transactionlog which is typically a media specific implementation associated withstorage devices. Many of these functions are described in more detailbelow or in Aasheim '672.

The media abstraction layer 306 isolates the entry logic layer 304 fromthe device specifics of the underlying data source 310. Consequently,file system 300 can easily be ported to entirely new types of device byimplementing a media abstraction layer 306 specific to the brand and/ortype of data source 310. Again, the data source 310 could be any type ofdata source, such as a hard disk, removable storage, network attachedstorage, wireless network, etc. This permits the entry logic layer 304to generically support calling applications without having to beconfigured or reconfigured to interface with lower level logic layers.

Media interface layer 308 is configured to interface directly with thedata source 310 and further isolates upper logic layers from theelectrical specifics of the data source. Accordingly, media interfacelayer 308 is configured to carry out the lowest of lower levelcommunications on behalf of file system 300 in response to datarequests. It is intended that the media interface layer 308 beprogrammably configurable by a user (i.e., OEM) to directly interfacewith the data source. Practically, this means that differentbrands/types of the same type of media can be supported without havingto make changes to the media abstraction layer 306. Thus, assuming thatthe data source is of the same type but of a different brand, then anOEM may only need to create a new or modified media interface layer 308.

Thus, file system 300 is media agnostic allowing OEMs to use the filesystem across a wide variety of different types of data sources 310.Generally, the entry logic layer 304 does not need to be configured tohave intimate knowledge of the operating characteristics of the datasource. Each of the interchangeable and customizable layers 304, 306,and 308, also permits OEMs to easily customize the different layers fortheir respective data sources.

File System Implementations for Flash

As mentioned above, the data source may include a variety of differenttypes and/or brands of data sources. Although many of the embodimentsdescribed below are described in conjunction with flash memory as thedata source, such is not required for the operation of the claimedinventions. Other various examples of data sources mentioned above willsuffice to implement many of the inventions describe herein, such as theoverall architecture of file system 104, and 300 described above withreference to FIGS. 1, 2 and 3.

Flash Medium Operating Characteristics

This discussion assumes that the reader is familiar with basic operatingprinciples of flash memory media. Nevertheless, a general introductionto two common types of nonvolatile random access memory, NAND and NORFlash memory media, is provided to better understand the exemplaryimplementations described herein. These two example flash memory mediawere selected for their current popularity, but their description is notintended to limit the described implementations to these types of flashmedia. Other electrically erasable and programmable read-only memories(EEPROMs) would work too. In most examples used throughout this DetailedDescription numbers shown in data structures are in decimal format forillustrative purposes.

FIGS. 4 and 5 illustrate logical representations of example NAND and NORflash memory media 400, 500, respectively. Both media have universaloperating characteristics that are common to each, respectively,regardless of the manufacturer. For example referring to FIG. 4, a NANDflash memory medium is generally split into contiguous blocks (0, 1,through N). Each block 0, 1, 2, etc. is further subdivided into Ksectors 402; standard commercial NAND flash media commonly contain 8,16, or 32 sectors per block. The amount of blocks and sectors can vary,however, depending on the manufacturer. Some manufacturers refer to“sectors” as “pages.” Both terms as used herein are equivalent andinterchangeable.

Each sector 402 is further divided into two distinct sections, a dataarea 403 used to store information and a spare area 404 which is used tostore extra information such as error correction code (ECC). The dataarea 403 size is commonly implemented as 512 bytes, but again could bemore or less depending on the manufacturer. Commonly, the size of thearea spare 404 is implemented as 16 bytes of extra storage for NANDflash media devices. Again, other sizes, greater or smaller can beselected. In most instances, the spare area 404 is used for errorcorrecting codes, and status information.

A NOR memory medium 500 is different than NAND memory medium in thatblocks are not subdivided into physical sectors. Similar to types of RAMdevices, each byte stored within a block of NOR memory medium isindividually addressable. Practically, however, blocks on NOR memorymedium can logically be subdivided into physical sectors with theaccompanying spare area.

Aside from the overall layout and operational comparisons, someuniversal electrical characteristics (also referred to herein as “memoryrequirements” or “rules”) of flash devices can be summarized as follows:

1. Write operations to a sector can change an individual bit from alogical ‘1’ to a logical ‘0’, but not from a logical ‘0’ to logical ‘1’(except for case No. 2 below);

2. Erasing a block sets all of the bits in the block to a logical ‘1’;

3. It is not generally possible to erase individual sectors/bytes/bitsin a block without erasing all sectors/bytes within the same block;

4. Blocks have a limited erase lifetime of between approximately 100,000to 1,000,000 cycles;

5. NAND flash memory devices use ECC to safeguard against datacorruption due to leakage currents; and

6. Read operations do not count against the write/erase lifetime.

Resident Flash File System

FIG. 6 shows a transactional file system 600 (“file system 600”)implemented for a flash memory media, such as those described in FIGS. 4and 5. Like file system 300, file system 600 includes threeinterchangeable and/or customizable logic layers 602 that operate aspart of a computer such as computer 100 shown in FIG. 1. In particular,file system 600 includes an entry logic layer 604, a media abstractionlayer 606, and a media interface layer 608. Although the exemplaryimplementation is shown to include these three elements, variousfeatures from each them may be selected to carry out some of the morespecific implementations described below. So while the describedimplementation shows three distinct layers of logic, many of thetechniques described below can be implemented without necessarilyrequiring all or a portion of the features from either layer of logic.Furthermore, the techniques may be implemented without having the exactdivision of responsibilities as described below

For a general understanding of the responsibilities of layers 604, 606,and 608 please refer to file system 300 described above. Otherwise, thefollowing discussions will primarily focus on features provided by mediaabstraction layer 606. Media abstraction layer 606 is responsible forgenerally performing logical-to-physical transformations, ensuring dataintegrity, transaction logging, wear-leveling, power-failure detectionand correction, and other related tasks. To accomplish these tasks, themedia abstraction layer is composed of several modules each responsiblefor performing their own task in conjunction with the other modules.

FIG. 7 shows modules 700 implemented for media abstraction layer 606.Modules 700 are comprised of a meta-information storage module 702,sector manager 704, a logical-to-physical sector mapping module 706, acompactor module 708, a power failure manager module 710, and atransaction log module 712. Functionality provided by each of themodules is not restricted to a file system and can be implementedseparately or in conjunction with a program application(s), an operatingsystem(s), or drivers.

Briefly, the meta-information storage module 702 stores files usingmeta-information to link the files if data associated with the filesspans over several sectors. The sector manager 704 provides a pointer toa sector available, i.e., “free” to receive new data. Thelogical-to-physical sector mapping module 706 manages data as it goesfrom a file system domain of logical sector addressing to a flash mediumdomain of physical sector addressing. The compactor module 708 providesa mechanism for clearing blocks of data (also commonly referred to inthe industry as “erasing”) to ensure that enough free sectors areavailable for writing data. Compactor module 708 permits file system 600perform uniform and even wear-leveling. Power failure manager module 710detects when a power failure event has occurred and takes correctionaction to ensure that data integrity is maintained by the file system.Transaction log module 712 is responsible for recording transactioninformation on the flash medium and permits the media abstraction layer606 to play back transactions in the event there is a power failureprior to completing the transactions. Aspects of each of these modules700 shall be described in more detail below.

Meta-information Storage

File system 600 (FIG. 6) does not use an allocation table to store thefile system's meta-information. Although file system 600 could beimplemented to do so, there are drawbacks to doing so: (1) Updating anallocation table unnecessarily wears down the life of a flash medium andcreates too many dirty sectors; and (2) each physical sector of flashmemory has a limited number of non-overlapping writes, consequently,there is no efficient guarantee that data integrity of individualsectors in the allocation table during a power-failure event. Mediaabstraction layer 606, through the meta-information storage module 702,overcomes these problems by storing data on the flash medium as a seriesof linked-lists.

FIG. 8 shows how meta-information storage module 702 stores data on theflash medium 800. Referring to FIG. 8, flash medium 800 represents anytype of flash medium. Physical sectors 802 are logically organized intotwo regions: a data region 804 for storing data, such as files,directories, attributes, etc., and a metadata area 806 of the flashmedium that can be used to store meta-information. In particular, data(represented as a shaded pattern) is stored in data region 804 andpointers 808, represented in this example as 2, 5, 9, and 10 are storedin metadata area 806. Each pointer 808 contains meta-information thatindicates the next physical sector containing valid data. For example,suppose a file uses five sectors to store data spanning physical sectors1, 2, 5, 9, and 10. Notice that within each physical sector, there is apointer indicating the next physical sector containing data. The chainof sectors ends with an “end of file” (EOF) indicator. In other words,the meta-information (e.g., 2, 5, 9, and 10) links the individualsectors together in a list referred to herein as a “linked list,” whichin the example of FIG. 8 is the combination of all pointers 808.

Using this directory structure, meta-information such as pointers 808,are stored inline with the data being written to the flash medium 800.This permits file system 600 to limit the number of non-overlappingwrite operations per sector. Additionally, the spare area 806 found inmost flash memory media can be used to store metadata bits to explicitlytrack write operations to either region, data 804 or spare area 806.Also by storing meta-data such as pointers 808 in line with data ratherthan in a table, it is possible to ensure the validity of data in asector during a power-failure event.

In the exemplary implementation, data region 804 and metadata region 806do not include the spare area of 404 of a NAND flash memory device,which is shown in FIG. 4. That is, the first region (data region 804)and the second region (metadata region 806) reside within the data area403 of the device. Nevertheless, metadata region 806 is not limited tothe data area 403 and can reside anywhere within a physical sectorincluding the spare area, such as spare area 404 of FIG. 4.

FIG. 9 illustrates a method 900 for a storing metadata in-line withdata. Method 900 includes blocks 902 and 904. The order in which themethod is described is not intended to be construed as a limitation.Furthermore, the method can be implemented by file system 600, but mayalso be implemented in any suitable hardware, software, firmware, orcombination thereof.

At block 902, data is stored in a first region of a physical sector of aflash medium. For example, data may be stored in a data region 804 (FIG.8) for storing data, such as files, directories, file attributes, andother file related items. At block 904, a pointer is stored in secondregion of the physical sector indicating a next physical sector in whichvalid data is located on the flash medium. For example, referring toFIG. 8, a pointer 2 is stored in the meta-information region (metadataarea) 806 of physical sector 1. Pointer 2 indicates the next physicalsector (physical sector 2 ) in which valid data is located on the flashmedium. In other words, pointer 2 links the location of data stored inphysical sector 1 to data stored in physical sector 2. Process 900 willnormally repeat itself until all data locations are linked together.

Tracking Data

File system 600 uses logical sector addressing to read and storeinformation on flash memory. Logical sector addresses are addresslocations that the file system reads and writes data to. They are“logical” because they are relative to the file system. In actuality,data may be stored in completely different physical locations on theflash memory. These physical locations are referred to as physicalsector addresses.

File system 600 is responsible for linking all logical sector addressrequests (i.e., read & write) to physical sector address requests. Theprocess of linking logical-to-physical sector addresses is also referredto herein as mapping. Going from logical to physical sector addressespermits file system 600 to have maximum flexibility when deciding whereto store data on the flash memory medium 601. Logical-to-physical sectormapping module 706 permits data to be flexibly assigned to any physicallocation on the flash memory medium, which provides efficiency for othertasks, such as wear-leveling and recovering from a power failure. Italso permits the file system 600 at entry level logic layer 604 to storedata (send data requests to media abstraction layer 606), withoutneeding intelligence to know that the data is actually being stored on aflash medium in a different fashion.

FIG. 10A shows an exemplary implementation of a data structure (i.e., atable) 1000A generated by the file system 600. The data structure 1000Ais stored in a volatile portion of memory 106 (see FIG. 1), such as sometype of random access memory (RAM). The data structure 1000A includesphysical sector addresses 1002 that have a corresponding logical sectoraddress 1004. An exemplary description of how table 1000A is generatedis described with reference to FIG. 11.

FIG. 11 illustrates a method 1100 used to track data on a flash memorymedium 601 when the file system 600 performs write requests. Method 1100includes blocks 1102-1118. The order in which the method is described isnot intended to be construed as a limitation. Referring to FIGS. 10A and11, at block 1102, media abstraction layer 606 receives a request towrite data to a specified logical sector address 1004.

At block 1104, the sector manager 704 ascertains a free physical sectoraddress location on the flash medium that can accept data associatedwith the write request (how the sector manager 704 chooses physicalsector addresses will be explained in more detail below). A freephysical sector is any sector that can accept data without the need tobe erased first. Once the sector manager 704 receives the physicalsector address associated with a free physical sector location, thelogical-to-physical sector mapping module 706 assigns the physicalsector address to the logical sector address 1004 specified by writerequest forming a corresponding relationship. For example, a physicalsector address of 0 through N can be assigned to any arbitrary logicalsector address 0 through N.

Next, at block 1106, the logical-to-physical sector mapping module 706stores the corresponding relationship of the physical sector address tothe logical sector address in a data structure, such as the exemplarytable 1000A in memory 106. As shown in the exemplary data structure1000A, three logical sector addresses 1004 are assigned to correspondingphysical sector addresses 1002.

Next, at block 1108 data associated with the logical sector addresswrite request is stored on the flash medium at the physical sectoraddress location assigned in block 1104. For example, data would bestored in physical sector address location of zero on the medium 601,which corresponds to the logical sector address of 11.

Now, at block 1110, suppose for example purposes another write requestis received by media abstraction layer 606, but in this case, to modifydata associated with a logical sector address previously issued in block1102. Then, file system 600 performs functionality described in blocks1112 through 1114, which are identical to steps 1104 through 1108,respectively, and are described above.

In block 1118, however, after the updated data associated withperforming block 1110 is successfully stored on the flash medium, thelogical-to-physical sector mapping module 706 marks the old physicalsector address assigned in block 1104 as “dirty.” Old data is markeddirty after new data is written to the medium 601, so in the event thereis a power failure in the middle of the write operation, thelogical-to-physical sector mapping module 706 will not lose old data. Itis possible to lose new or updated data in blocks 1102 or 1110, butsince there is no need to perform an erase operation only one item ofnew or modified data is lost in the event of a power failure.

FIG. 10B shows a data structure 1000B which is the same as datastructure 1000A, except its contents have been updated. In this examplethe file system 600 has updated data associated with logical sectoraddress 11. Accordingly, file system 600 reassigns logical sectoraddress 11 to physical sector address 3 and stores the reassignedcorresponding relationship between the these two addresses in datastructure 1000B. As illustrated in data structure 1000B, the contents oflogical sector 11 are actually written to physical sector address 3 andthe contents of sector 0 are marked “dirty” after the data contents aresuccessfully written into physical sector address 3 as was describedwith reference to blocks 1110-1118.

This process of reassigning logical-to-physical sector address whenpreviously stored data is updated by the file system 600, permits writeoperations to take place without having to wait to move an entire blockof data and perform an erase operation. So, method 1100 permits the datastructure to be quickly updated and then the physical write operationcan occur on the actual physical medium. Media abstraction layer 606uses the data structures, such as 1000A/1000B, to correctly maintainlogical-to-physical mapping relationships.

When there is a read request issued by or to files system 600, the mediaabstraction layer 606 via logical-to-physical mapping module 706,searches the data structure (such as 1000A or 1000B) to obtain thephysical sector address which has a corresponding relationship with thelogical sector address associated with read request. Media abstractionlayer 606 then uses that physical sector address as a basis to send dataassociated with the read request back to entry logic layer 604.Accordingly, entry logic layer 604 does not need intelligence to knowthat its requests to logical sector addresses are actually mapped tophysical sector addresses.

Write Pointer and Continuous Circle

FIG. 12 is a diagram of flash memory medium 601 viewed and/or treated asa continuous circle 1200 by the file system 600. Physically the flashmemory media is the same as either media shown in FIGS. 4 and 5 forinstance, except media abstraction layer 606 organizes the flash memorymedium as if it is a continuous circle 1200, containing 0-to-N blocks.Accordingly, the highest physical sector address (individual sectors arenot shown in FIG. 12 to simplify the illustration, but may be seen inFIGS. 4 and 5) within block N and the lowest physical sector addresswithin block 0 are viewed as being contiguous.

FIG. 13 illustrates another view of media 601 viewed as a continuouscircle 1200. In this exemplary illustration, the sector manager 704maintains a write pointer 1302, which indicates a next available freesector to receive data on the medium. The next available free sector isa sector that can accept data without the need to be erased first in aprescribed order. The write pointer 1302 is implemented as a combinationof two counters: a sector counter 1306 that counts sectors and a blockcounter 1304 that counts blocks. Both counters combined indicate thenext available free sector to receive data.

In an alternative implementation, the write pointer 1302 can beimplemented as a single counter and indicate the next physical sectorthat is free to accept data during a write operation. According to thisimplementation, the sector manager 704 maintains a list of all physicalsector addresses free to receive data on the medium. The sector manager704 stores the first and last physical sector addresses (the contiguousaddresses) on the medium and subtracts the two addresses to determine anentire list of free sectors. The write pointer 1302 then advancesthrough the list in a circular and continuous fashion. This reduces theamount of information needed to be stored by the sector manager 704.

FIG. 14 illustrates a method 1400 used by the sector manager 704 todetermine the next available free sector location for the file system600 to store data on the flash medium. Method 1400 also enables thesector manager 704 to provide each physical sector address (for the nextfree sector) for assignment to each logical sector address write requestby file system 600 as described above. Method 1400 includes blocks1402-1418. The order in which the method is described is not intended tobe construed as a limitation. Furthermore, the method can be implementedin any suitable hardware, software, firmware, or combination thereof.

At block 1402, the X block counter 1304 and Y sector counter 1306 areinitially set to zero. At this point it is assumed that no data resideson the medium 601.

At block 1404, media interface layer 608 receives a write request andthe sector manager 704 is queried to send the next available freephysical sector address to the logical-to-physical sector mapping module706.

At block 1406, the data is written to the sector indicated by the writepointer 1302. Since both counters are initially set to zero in thisexemplary illustration, suppose that the write pointer 1302 points tosector zero, block zero.

At block 1408, the sector counter 1306 is advanced one valid sector. Forexample, the write pointer advances to sector one of block zero,following the example from step 1406.

Next, in decisional block 1410, the sector manager 704 checks whetherthe sector counter 1306 exceeds the number of sectors K in a block. Ifthe Y count does not exceed the maximum sector size of the block, thenaccording to the NO branch of decisional block 1410, blocks 1404-1410repeat for the next write request.

On the other hand, if the Y count does exceed the maximum sector size ofthe block, then the highest physical sector address of the block waswritten to and the block is full. Then according to the YES branch ofblock 1410, the Y counter is reset to zero as indicated at block 1412.Next, in block 1414, X block counter 1304 is incremented by one, whichadvances the write pointer 1302 to the next block at the lowest validphysical sector address, zero, of that block.

Next, in decisional step 1416, the compactor module 708 checks whetherthe X block counter is pointing to a bad block. If it is, X blockcounter 1304 is incremented by one. In one implementation, the compactor406 is responsible for checking this condition. As mentioned above, thesector manager stores all of the physical sector addresses that are freeto handle a write request. Entire blocks of physical sector addressesare always added by the compactor during a compaction or duringinitialization. So, the sector manager 704 does not have to check to seeif blocks are bad, although the sector manager could be implemented todo so. It should also be noted that in other implementations block 1416could be performed at the start of method 1400.

In block 1417, the X block counter 1304 is incremented until it ispointing to a good block. To avoid a continuous loop, if all the blocksare bad, then method 1400 stops at block 1416 and provides an indicationto a user that all blocks are bad.

Next in decisional block 1418, the sector manager checks whether the Xblock counter 1304 exceeds the maximum numbers of blocks N. This wouldindicate that write pointer 1302 has arrived full circle (at the top ofcircle 1200). If that is the case, then according to the YES branch ofblock 1418, method 1400 repeats and the X and Y counter are reset tozero. Otherwise, according to the NO branch of block 1418, method 1400returns to block 1404 and proceeds.

In this exemplary method 1400, the write pointer 1302 initially startswith the lowest physical sector address of the lowest addressed block.The write pointer 1302 advances a sector at a time through to thehighest physical sector address of the highest addressed block and thenback to the lowest, and so forth. This continuous and circular method1400 ensures that data is written to each sector of the medium fairlyand evenly. No particular block or sector is written to more than anyother, ensuring even wear-levels throughout the medium. Accordingly,method 1400 permits data to be written to the next available free sectorextremely quickly without expensive processing algorithms used todetermine where to write new data while maintaining even wear-levels.Such conventional algorithms can slow the write speed of a computerdevice.

In an alternative implementation, it is possible for the write pointer1302 to move in a counter clock wise direction starting with highestphysical sector address of the highest block address N and decrement itscounters. In either case, bad blocks can be entirely skipped and ignoredby the sector manager. Additionally, the counters can be set to anyvalue and do not necessarily have to start with the highest or lowestvalues of for the counters.

FIG. 15 illustrates another view of media 601 viewed as a continuouscircle 1200. As shown in FIG. 15, the write pointer 1302 has advancedthrough blocks 0 through 7 and is approximately half way through circle1200. Accordingly, blocks 0 through 7 contain dirty data. That is, eachgood sector in blocks 0 through 7 is not free, and therefore, notavailable to receive new or modified data. Arrow 1504 represents thatblocks 0 through 7 contain used sectors. Eventually, the write pointer1302 will either run out of free sectors to write to unless sectors thatare marked dirty or are not valid are cleared and recycled. To clear asector means that sectors are reset to a writable state or in otherwords are “erased.” In order to free sectors it is necessary to erase atleast a block at a time. Before a block can be erased, however, thecontents of all good sectors are copied to the free sectors to adifferent portion of the media. The sectors are then later marked“dirty” and the block is erased.

The compactor module 708 is responsible for monitoring the condition ofthe medium to determine when it is appropriate to erase blocks in orderto recycle free sectors back to the sector manager 704. The compactormodule 708 is also responsible for carrying out the clear operation. Tocomplete the clear operation, the compactor module 708 like the sectormanager 704, maintains a pointer. In this case, the compactor module 708maintains a clear pointer 1502, which is shown in FIG. 15. The clearpointer 1502 points to physical blocks and as will be explained enablesthe compactor module 708 to keep track of sectors as the medium asblocks are cleared. The compactor module 708 can maintain a pointer to ablock to compact next since an erase operation affects entire blocks.That is, when the compactor 406 is not compacting a block, the compactormodule 708 points to a block.

In the event of power failure, the media abstraction layer 606 containssimple coded logic that scans the flash memory medium 601 and determineswhat locations are marked free and dirty. The logic is then able todeduce that data 1504 resides between the locations marked free anddirty. A head (write pointer) and tail (clear pointer 1502) of the data(also referred to as a data stream 1504) is easily determined bylocating the highest of the physical sector addresses containing datafor the head and by locating the lowest of the physical sector addressescontaining data for the tail.

Power Failure Detection and Correction

A power failure can detrimentally affect the data (e.g. files) integrityof a file system, in a computer that uses flash media to store the data.For instance, suppose that a user of a computer is attempting to storedata and has just performed a task that issues a data request. In thisscenario the data request is to write data to the flash medium, butunfortunately the user accidentally drops the computer disconnecting thepower source. When the user reconnects the battery, will the file systemknow that a power failure event occurred? What will happen to the datathat the user was attempting to store? Did the data get stored on themedium? Does the file system know that the data was stored on themedium? Perhaps the data was just partially stored? How will the filesystem know whether more data was expected? How will the file systemrecover from the power failure event and still preserve data integrityfor the computer? How will the file system recover from the powerfailure event quickly and efficiently?

The media abstraction layer 606 can be implemented with functionality todetect a power failure event after a computer is initialized (e.g., thecomputer is turned-on from a powered-off state or rebooted). The mediaabstraction layer 606 can also be implemented with functionality tocorrect for corrupt data such as partial data stored on the flash mediumas a result of write operation interrupted by a power failure event. Inparticular, the power failure manager module 710 efficiently detects apower failure event. It also ensures that data integrity is preserved,even for the last physical sector to receive data just prior to thepower failure event, including any accompanying meta-information.

FIG. 16 shows how power failure manager module 710 storesmeta-information 1602 on a flash medium 1600 when a computer shuts-down(e.g., powers-off or reboots) according to a normal shutdown mode. Inthe exemplary implementation, the meta-information 1602 is a bit in thespare area designated as the “shutdown bit”, which could by any bit1-to-N. Alternatively, the meta-information 1602 could take other formssuch as more than one bit and could be stored in other regions of thephysical sector.

When the computer 100 shuts-down in a normal shutdown mode (as opposedto shutting abnormally because of a power failure event), the powerfailure manager module 710 stores meta-information 1602 in a physicalsector indicated by write pointer 1302. The location of the writepointer 1302 is relative and advances in circular fashion as describedabove. Accordingly, the meta-information 1602 can be stored in any validphysical sector of the flash medium indicated by the write pointer atthe time of a normal shutdown.

Now, when the computer 100 initializes, the power failure manager module710 simply scans the physical sector indicated by the write pointer 1302to ascertain whether the meta-information 1602 is present. If themeta-information 1602 is present, the computer previously shutdownaccording a normal shut down mode. If the meta-information is not found,then a conclusion is made by the power-failure manager 710 that thecomputer did not shut-down properly due to some type of catastrophicevent such as power failure.

FIG. 17 is a method 1700 for detecting power failure. Method 1700includes blocks 1702 through 1708. The order in which the method isdescribed is not intended to be construed as a limitation. Furthermore,the method can be implemented in any suitable hardware, software,firmware, or combination thereof.

At block 1702, meta-information such as a shut-down bit is stored at alocation on a flash medium indicated by a write pointer if a computershuts-down according to a normal shutdown mode. For example, if computer100 shown in FIG. 16 shuts down normally, then the media abstractionlayer 606 will store meta-information in the spare area of a physicalsector indicated by write pointer 1302 of the flash medium 1600. At ablock 1703, the computer 100 is initialized or rebooted. The computermay have been reactivated after a power-failure event, system reboot, orany other situation where the computer is deactivated.

At a decisional block 1704, during initialization of the computer, acheck is made whether the meta-information is present in the location onthe flash medium indicated by the write pointer. For example, referringto FIG. 16, when computer 100 (shown in FIG. 1) is rebooted orturned-on, the media abstraction layer 606 checks the physical sectorindicated by the write pointer 1302 for the meta-information 1602.

Referring back to FIG. 17, if according to the Yes branch of decisionalblock 1704 the meta-information is present in the physical sectorindicated by the write pointer, then according to block 1706 aconclusion is made that the computer shutdown according to a normal modeand no corrective action is needed.

On the other hand, if according the No branch of decisional block 1706the meta-information is not present in the physical sector indicated bythe write pointer 1302, then according to block 1708 a conclusion ismade that the computer did not shutdown according to the normal shutdownmode, e.g., a power failure event occurred.

With respect to a power-failure, the media abstraction layer 606 is alsoresponsible for ensuring that data integrity is maintained on a physicalsector basis. As described above, power failure can occur at any timeand to ensure data integrity, it is important to prevent partial sectorwrites. Accordingly, FIG. 18 shows how power failure manager module 710stores meta-information 1802 on the flash medium 1800 to safeguardagainst a partial sector write. In this example, power failure managermodule 710 stores meta-information in the form of a “data valid bit.”The “data valid bit” could be any bit 1-to-N in the spare area that isset active after completion of each write operation to a particularsector. Alternatively, the meta-information 1602 could take other formsand could be stored in other regions of the physical sectors.

Typically, the data valid bit is the last bit “N” in the spare area of aphysical sector to be written after completion of a write operation.Because the data valid bit is the last bit written in each physicalsector, if the bit is not found in a physical sector indicated by thewrite pointer, then the media abstraction layer 606 can conclude thatthe physical sector contains corrupt data. As a result, the mediaabstraction layer 606 marks the sector as “dirty.” File system 600 willnot use the data in this physical sector. Again, the write pointer 1302is used to locate the last sector written to prior to initialization.

The media abstraction layer 606, by using the data valid bitmeta-information, ensures a physical sector is not corrupt due to anincomplete write. Hence, data integrity at the sector level isguaranteed in the event of a power failure event. Moreover, the “dirtysector” will later be reclaimed by the compactor and no additionalclean-up is necessary. Thus, the amount of data potentially lost duringa power-failure event is minimized to a single sector.

FIG. 19 is a method 1900 for ensuring data integrity in the event ofpower-failure event. Method 1900 includes blocks 1902-1916. The order inwhich the method is described is not intended to be construed as alimitation. Furthermore, the method can be implemented in any suitablehardware, software, firmware, or combination thereof.

At block 1902, a write pointer is used to advance through a circularsequence of memory locations in the flash medium indicating a memorylocation that is available to receive data as the write pointeradvances. At block 1904, data is written into the memory location (suchas a physical sector) indicated by the write pointer.

At block 1906, meta-information (also referred to as metadata) iswritten into the memory location indicated by the write pointer as thelast item written into the physical sector. For example, a data validbit is written into the spare area of a physical sector. Iffunctionality performed in blocks 1904 or 1906 is interrupted prior tobe completed, then the meta-information will not have a chance to bewritten into the memory location. This will serve as an indicator thatthere is corrupt data in the memory location, because either no or onlypartial data has been written into the memory location prior to theoccurrence of the interrupting event.

At block 1908, the computer is initialized (e.g., powered-on orrebooted) and the file system scans the media for the write pointer. Atdecisional block 1910, the memory location indicated by the writepointer is searched to determine whether the meta-information, such asthe data valid bit is present.

If according to the Yes branch of decisional block 1910, themeta-information is located in the memory location indicated by thewrite pointer, then according to block 1912 the memory location (e.g.,physical sector) is deemed to include valid data. If according to the Nobranch of decisional block 1910, meta-information is not located inmemory location indicated by the write pointer, then according to block1914 the memory location is deemed to include partial or invalid dataand the memory location is marked “dirty.”

Alternatively, the data valid bit can also be used to detect a powerfailure. That is if the data valid bit is not located in the physicalsector indicated by the write pointer after computer 100 is initialized,then the media abstraction layer 606 can deduce that a power failureevent occurred.

Transaction Support, Transaction Log

To ensure that file system 600 maintains consistency during apower-failure event, the media abstraction layer 606 can be implementedto record and maintain a transaction log. In particular, the transactionlog module 712 can be configured to maintain a log of transactionsperformed by the file system 600. A transaction log generally serves asa history journal of transactions performed by the file system over aperiod of time that can be read back in the event there is a powerfailure to ensure integrity of directories stored on the flash medium.These transactions include creating files, deleting files, copyingfiles, reading files, creating directories, creating subdirectories, andother related file tasks.

Most current file systems reserve a fixed region of the flash medium tostore log information. For example, a file system may reserve 10 MB ofthe flash medium at a particular fixed region on the flash mediumexclusively for storing transaction log data. Unfortunately, this designhas several drawbacks. Reserving too little space for the fixed regioncan slow down the speed of the file system, because only a finite numberof user requests can operate concurrently. On the other hand, reservingtoo much space impacts the amount of user data that can be stored on themedia. Additionally, fixing the transaction log in a particular regionof the flash memory can create a host of problems, the worst being thatthis fixed flash region has a greater potential for going “bad” or beingworn out over time, which can cause a catastrophic failure for theentire flash medium.

To overcome these problems, the media abstract layer 606 permitstransaction logs to be placed at arbitrary places on the flash medium.For example, FIG. 20 shows transaction information 2002 stored at anarbitrary location 2004 on a flash medium 2000 indicated by writepointer 1302. As used herein “arbitrary location” means any arbitraryphysical sector address associated with the physical sectors in whichtransaction information 2002 is stored and includes potentially allvalid physical sector addresses on the flash medium. In other words, oneor more transaction logs are not restricted to a fixed region ofphysical sector addresses on the flash medium. Although reference may bemade herein to a singular transaction log for discussion purposes only,it is possible to store multiple transaction logs and different types oftransactions logs at arbitrary locations on the flash medium.

When the media abstraction layer 606 receives requests to perform a filetransaction, the transaction log module 712 stores transactioninformation associated with performing the file transaction at the nextavailable free physical sector on the medium. In other words, thetransaction log module 712 stores transaction information 2002associated with performing a file request at the physical sector addresson the medium indicated by the write pointer 1302. The transactioninformation includes operation code 2022, transaction data 2024, andmetadata 2008.

Operational code 2022 is typically stored at the beginning of the dataportion of a physical sector, although the operational code 2022 couldbe stored at other locations within the physical sector. The operationalcode 2022 is a number used to indicate what type of action is associatedwith a pending transaction (i.e., delete file, create file, open file,etc.). In the exemplary implementation, the operational code is fourbytes, but could include more than four bytes if necessary. The rest ofthe data 2024 in the physical is associated with the particularities ofthe transaction such as the name of directory or file, and other morespecific information.

The transaction information 2002 further includes metadata 2008indicating that the arbitrary physical sector address containstransaction information 2002. The metadata 2008 in the exemplaryimplementation is stored in the spare area of the flash medium, butcould be stored in other regions of the physical sector. In particular,transaction log metadata could be any of the one or more bits 0, 1, 2.,. . . , N in the spare area of the physical sector used to uniquelyidentify the physical sector as containing transaction information 2002.Thus, the transaction log metadata 2008 permits any physical sector(s)to be readily identified from other physical sector containing data, asa physical sector that contains transaction information.

Thus, file system 600 can identify the arbitrary physical sectoraddresses that contain transaction information 2002, from thetransaction log metadata 2008. The file system can identify the physicalsector addresses at initialization. The media abstraction layer 606scans the spare areas and checks for the presence of transaction logmetadata 2008 to identify and locate which physical sector addressescorresponding to physical sectors contain transaction information 2002.

Accordingly, the amount of space needed for transaction logs is notfixed. File system can continue to create transaction logs if there isfree space on the medium. Thus, size requirements for transaction logsdynamically adapt to the run-time behavior of the file system. The mediaabstraction layer is able to avoid the host of problems mentioned aboveby storing transaction information at arbitrary locations. At boot time(e.g., initialization), no more time is needed to identify sectorscontaining transaction logs, because the spare area is scanned for otherinformation relevant to other parameters.

Open transaction logs can be “closed” after a transaction issuccessfully completed by marking the physical sector containing thetransaction data dirty. For example, when the file system receives anindication that a transaction has been completed by the media interfacelayer 608, the physical sector is marked dirty and will later berecycled by the compactor module 708. If after initialization, opentransaction logs are located, then file system 600 deduces that thesetransactions were not performed. Accordingly, file system 600 completesall pending transactions stored in physical sectors not marked “dirty”and the one or more transaction logs associated with transactionoperations are closed. This is referred to as transaction log playbackand is particularly useful to ensure data integrity after apower-failure event.

FIG. 21 shows a method 2100 for implementing a transaction log and theability to recover transactions in the event of a power-failure eventthrough transaction playback. Method 2100 includes blocks 2102 through2114. The order in which the method is described is not intended to beconstrued as a limitation. Furthermore, the method can be implemented inany suitable hardware, software, firmware, or combination thereof.

In block 2102, a transaction log is stored at an arbitrary physicalsector address of flash medium. For example, transaction informationassociated with performing a file request is stored at any locationindicated by the write pointer. This transaction log and other potentialtransaction logs are not restricted to fixed locations on the medium.

In block 2104, metadata is written into a spare area of the physicalsector containing the transaction information. The metadata indicatesthat the physical sector contains transaction information. The metadatacan take many forms and is not necessarily restricted to the spare areaof the physical sector.

In block 2106, the physical sector containing the transactioninformation is marked “dirty” when the file system completes thetransaction associated with the transaction information stored in thesector. If the transaction is not completed then the physical sector isnot marked dirty. In block 2108, the computer is initialized. Thecomputer may have been reactivated after a power-failure event, systemreboot, or any other situation where the power source to the computerwas deactivated.

In a decisional block 2110, all the physical sectors not marked dirtyare scanned to determine whether any file requests are pending by virtueof the metadata.

If according to the Yes branch of decisional block 2110 one or more filerequests are found to be pending, then in block 2112 the file systemrecognizes that the one or more transactions stored in each respectivephysical sector is pending and has not been completed. Accordingly, thefile system can schedule that the data request be attempted again (inthe event there was a power failure event that interrupted the datarequest from being accomplished).

If according to the No branch of decisional block 2110 no requests arelocated, then according to block 2114 a conclusion is made that the alltransactions were completed prior to initialization.

Free Space Management

FIG. 22 illustrates a dynamic look-up data structure 2200 to track datastored in the flash memory medium. Data structure 2200 includes a masterdata structure 2202, one or more secondary data structures 2204, 2206,and one or more counters 2208, 2210.

The data structures and counters are generated and maintained by filesystem 600, but can also be generated and maintained by other softwaresystems including the operating system, program applications, drivers,and other related systems. The data structures and counters are storedin a volatile portion of memory 106 (referring to computer 100 of FIG.1), but could also be stored in non-volatile memory.

The one or more secondary data structures 2204, 2206 contain mappings oflogical-to-physical sector addresses. Each of the secondary datastructures 2204, 2206 has a predetermined capacity of mappings. Themaster data structure 2202 contains a pointer to each of the one or moresecondary data structures 2204, 2206. Each secondary data structure isallocated on an as needed basis for mapping those logical-to-physicaladdresses that are used to store data. Once the capacity of a secondarydata structure 2204, 2206, et cetera, is exceeded another secondary datastructure is allocated, until eventually all possible physical sectoraddresses on the flash medium are mapped to logical sector addresses.Each time a secondary table is allocated, a pointer contained in themaster data structure 1002 is enabled by the file system 600 to point toit.

Each counter 2208, 2210 is associated with a respective secondary datastructure 2204, 2206. Each counter 2208, 2210, provides an indication ofwhen each of the one or more secondary data structures 2204, 2206,respectively, reaches the predetermined capacity of mappings. Forexample, in one implementation suppose that each secondary datastructure can map K logical-to-physical sector addresses. Accordingly,each counter (such as 2208, 2210) is initialized to K when theassociated secondary data structure is created (e.g., such as 2204,2206, respectively). Now, each time the file system writes to a newlogical sector address in one of the secondary data structures, thecounter associated with that secondary data structure is decremented. Onthe other hand, each time the file system indicates that a logicalsector address is no longer needed (such as delete file operation) thecounter associated with that secondary data structure is incremented.Alternatively, the counter could be initialized at zero and incrementedeach time a logical sector address is added and decremented each time alogical sector address is removed. In this alternative counting method,when the counter reaches K, this would indicate that the secondary datastructure is full and an additional secondary data structure needs to beallocated.

The media abstraction layer 606 can use the counters (count variablesmaintained by each of the counters) to track free sectors. If the mediaabstraction layer 606 finds that a count variable equals zero for aparticular counter, then the media abstraction layer 606 does not needto search through the secondary mappings in the associated secondarydata structure for a free location when creating a new logical sectoraddress. The counter permits the media abstraction layer 606 to avoidunnecessarily searching a secondary data structure to find a freelogical sector when the counter indicates that the secondary datastructure is full, i.e., reached the predetermined capacity of mappingsfor that secondary data structure. As the number of logical-to-physicalsector address mappings increases, the counters will increasingly assistin reducing search space required by the media abstraction layer. Thecounters will also speed up the process of locating free space andallocating secondary data structures.

Accordingly, file system 600 dynamically allocates one or more secondarydata structures based on the amount of permanent data stored on theflash medium itself. The size characteristics of the secondary datastructures are computed at run-time using the specific attributes of theflash memory medium. Secondary data structures are not allocated unlessthe secondary data structure previously allocated is full orinsufficient to handle the amount of logical address space required bythe file system. Dynamic look-up data structure 2200, therefore,minimizes usage of memory. Dynamic look-up data structure 2200 lendsitself to computer devices 100 that use calendars, inboxes, documents,etc. where most of the logical sector address space will not need to bemapped to a physical sector address. In these applications, only afinite range of logical sectors are repeatedly accessed and new logicalsectors are only written when the application requires more storagearea.

The master data structure 2202 contains an array of pointers, 0 throughN that point to those secondary data structures that are allocated. Inthe example of FIG. 22, the pointers at location 0 and 1 point tosecondary data structures 2204 and 2206, respectively. Also, in theexample illustration of FIG. 22, pointers 2 through N do not point toany secondary data structures and would contain a default setting,“NULL”, such that the logical-to-physical sector mapping module 706knows that there are no further secondary data structures allocated.

In each secondary data structure 2204, 2206 only a portion of the totalpossible medium is mapped in the secondary data structures. Thesecondary data structures permit the file system to reduce the amountspace needed in memory to only those portions of logical sectorsaddresses issued by the file system. Each secondary data structure is(b*k) bytes in size, where k is the number of physical sector addressescontained in the data structure and b is the number of bytes used tostore each physical sector address.

FIG. 23 illustrates a method 2300 for dynamically allocating look-updata structures for tracking data on the flash memory medium. Method2300 includes blocks 2302 through 2312. The order in which the method isdescribed is not intended to be construed as a limitation. Furthermore,the method can be implemented in any suitable hardware, software,firmware, or combination thereof.

In block 2302, a master data structure containing the pointers to one ormore secondary data structures is allocated and/or maintained. Themaster data structure in this exemplary implementation is fixed in size.At the time a computer initializes, the size of the flash memory mediumused to store data is determined. Based on the size of the flash medium,the media abstraction layer calculates a range of physical addresses.For example, suppose the size of the flash medium is 16 MB, then a NANDflash medium 400 will typically contain 32768 sectors each 512 bytes insize. This means that the media abstraction layer 606 may need to map atotal of 0 through 32768 logical sectors in a worse case scenario,assuming all the memory space is used on the flash medium. Knowing thatthere are 2¹⁵ sectors on the medium, the media abstraction layer 606 canuse 2 bytes to store the physical sector address for each logical sectoraddress. So the master data structure is implemented as an array of 256DWORDs (N=256), which covers the maximum quantity of logical sectoraddresses (e.g., 32768) to be issued by the files system. So, there area total of 256 potential secondary data structures.

In block 2304 the secondary data structure(s) are allocated and/ormaintained. First, a determination is made to calculate the smallestpossible size for each potential secondary data structure. With respectto the example above and using simple division, 32768/256=128 logicalsector addresses are supported by each data structure. As mentionedabove, the entire physical space can be mapped using 2 bytes, b=2,therefore, each secondary data structure will be 256 bytes in size or(b=2*k=128). This will also serve as a predetermined capacity of totalmappings each secondary data structure can store.

Now, knowing the size of each secondary data structure, suppose that thefile system requests to write to logical sector addresses 50-79, alsoknown as LS50-LS79. To satisfy the write requests a calculation is madethat the first pointer in master data structure is used for logicalsector addresses LS0-LS127 associated with a first secondary datastructure which is 256 bytes in size in memory. For example, the firstpointer in position zero of the master data structure 2202 shown in FIG.22 is used to point to a secondary data structure such as secondary datastructure 2204.

In block 2306, at least one counter is allocated or maintained for eachsecondary data structure allocated in block 2304. The counter providesan indication of when each associated secondary data structure reachesits predetermined capacity of mappings. This can be accomplished severaldifferent ways. For example, assuming the secondary data structure 2204can map a maximum of K logical-to-physical addresses, then counter 2208is initialized to K when the secondary data structure 2204 is allocated.Each time data is written to a new logical sector address, the count isdecremented. Each time a logical sector address is no longer needed, thecount is incremented.

Each time new data is added to the secondary data structure(s), inaccordance with descional block 2308 the counter associated withsecondary data structure to receive the new data is checked to ascertainif the counter indicates that the secondary data structure reached thepredetermined capacity. For example, if the count variable for counter2208 in FIG. 22 indicates zero, then secondary data structure 2204 hasreached capacity and there is no need to search the data structure 2204for free space to store data.

If the secondary data structure has reached it predetermined capacity asindicated by the counter 2208, then according to the Yes branch ofdecisional block 2308, method 2300 repeats itself, except (i) anadditional pointer is activated in master data structure 2202 unlessmaster data structure 2202 is full; (ii) an other secondary datastructure is allocated, and (iii) another counter is allocated andinitialized to K and monitored.

If the secondary data structure has not reached capacity as indicated bythe counter, then according to the No branch of decisional block 2308,process 2300 checks whether the secondary data structure is emptyaccording to decisional block 2310.

If according to the Yes branch of decisional block 2310, the secondarydata structure is empty then process 2300 proceeds to block 2312. Forexample, suppose the count variable for counter 2210 in FIG. 22 equalsK+1, because files or data had previously been deleted from secondarydata structure 2206. Then according to block 2312, at least onesecondary data structure is removed from memory creating more freespace. Then, method 2300 repeats itself, except (i) a pointer isdeactivated in master data structure 2202 unless there are no moresecondary data structures; and (ii) the counter associated withsecondary data structure that was removed is also deactivated.

If according to the No branch of decisional block 2310, the secondarydata is structure is not empty as indicated by the counter, then process2300 repeats itself.

Thus, using counters in conjunction with a dynamic lookup data structure2200 permits a file system or the like to allocate secondary datastructures based on a counter variable maintained for each secondarydata structure. This dramatically reduces the amount of memory requiredto maintain conventional bit map tables.

Conclusion

Although the invention has been described in language specific tostructural features and/or methodological acts, it is to be understoodthat the invention defined in the appended claims is not necessarilylimited to the specific features or acts described. Rather, the specificfeatures and acts are disclosed as exemplary forms of implementing theclaimed invention.

1. One or more computer-readable media comprising a file system that isexecutable by a computer, the file system comprising: a plurality ofinterchangeable and/or customizable logic layers configured to manageoperations on behalf of an application when making data requests to adata source, wherein functionality associated with the file system isaugmentable by substituting and/or modifying one or more of theplurality of interchangeable and/or customizable logic layers,respectively, enabling the file system to function with many differenttypes of data sources, wherein one of the plurality of interchangeableand/or customizable logic layers comprises a media abstraction layerconfigured to (i) manage input and output requests received from theapplication; (ii) perform logical to physical transformations; (iii)detect a power failure event; (iv) maintain data integrity and relatedmeta-information in the event of detecting a power failure event,wherein the data integrity and the related meta-information aremaintained for a last physical sector to receive data immediately priorto the power failure event, and (v) maintain a transaction log in arandom location.
 2. The file system as recited in claim 1, wherein themedia interface layer is further configured to interface directly withthe data source.
 3. The file system as recited in claim 1, wherein themedia interface layer is further configured to carry out a lowest oflower level communications on behalf of at least one of the plurality ofinterchangeable logic and/or customizable layers in response to datarequests.
 4. The file system as recited in claim 1, wherein the datasource is one of the following devices: fixed media, storage cards,network devices, networks, network storage and hard disks.
 5. The filesystem as recited in claim 1, wherein the data source is a physicaland/or virtual object that is configured to read and write data.
 6. Thefile system as recited in claim 1, wherein the data source includes oneof the following: a flash memory, a hard disk, a removable storagemedium, a network, and a communication device.
 7. The file system asrecited in claim 1, wherein the data requests include reading data fromthe data source and/or writing data to the data source.
 8. The filesystem as recited in claim 1, wherein one of the plurality ofinterchangeable and/or customizable logic layers comprises an entrylogic layer comprising a set of Application Protocol Interfaces (APIs)accessible by the application.
 9. The file system as recited in claim 1,wherein the computer is portable electronic device.
 10. The file systemas recited in claim 1, wherein one of the plurality of interchangeableand/or customizable logic layers comprises a media inter-face layerconfigured to interface directly with a flash medium.
 11. One or morecomputer-readable media comprising a file system that is executable by acomputer, the file system comprising: a plurality of interchangeableand/or customizable logic layers configured to manage operations onbehalf of an application when making data requests to a data source, theplurality of interchangeable and/or customizable logic layerscomprising: an entry logic layer including a set of Application ProtocolInterfaces (APIs) accessible by the application; a media abstractionlayer configured to manage data requests made by the application;perform logical to physical transformations; detect and manage a powerfailure event; maintain data integrity and related meta-information forthe file system in the event of a power failure event; and, maintain atransaction log in a random location, wherein the data integrity and therelated meta-information are maintained for a last physical sector toreceive data immediately prior to the power failure event; and a mediainterface layer, configured to carry out a lowest of the lower levelcommunications on behalf of the entry logic layer and/or mediaabstraction layer in response to data requests.
 12. The file system asrecited in claim 11, wherein functionality associated with the filesystem is augmentable by substituting and/or modifying one or more ofthe plurality of interchangeable and/or customizable logic layers,respectively, enabling the file system to function with many differenttypes of data sources.
 13. The file system as recited in claim 11,wherein the media inter-face layer is configured to communicate directlywith the data source.
 14. The file system as recited in claim 11,wherein the media inter-face layer is programmably configurable by auser to directly interface with a data source.
 15. The file system asrecited in claim 11, wherein the data source is one of the followingdevices: fixed media, storage cards, network devices, networks, networkstorage and hard disks.
 16. The file system as recited in claim 11,wherein the data source is a physical and/or virtual object that isconfigured to read and write data.
 17. The file system as recited inclaim 11, wherein the data source includes one of the following: a flashmemory, a hard disk, a removable storage medium, a network, and acommunication device.
 18. The file system as recited in claim 11,wherein the data requests include reading data from the data sourceand/or writing data to the data source.
 19. A computer, comprising:processor a file system having a plurality of interchangeable and/orcustomizable logic layers configured to manage operations on behalf ofan application operating on the computer when making data requests to adata source, wherein functionality associated with the file system isaugmentable by substituting and/or modifying one or more of theplurality of interchangeable and/or customizable logic layers,respectively, enabling the file system to function with many differenttypes of data sources, wherein the data requests include reading datafrom the data source and writing data to the data source, wherein one ofthe plurality of interchangeable and/or customizable logic layerscomprises a media abstraction layer configured to (i) manage input andoutput requests received from the application; (ii) perform logical tophysical transformations; (iii) detect a power failure event, (iv)maintain data integrity and related meta-information in the event ofdetecting a power failure event, wherein the data integrity and therelated meta-information are maintained for a last physical sector toreceive data immediately prior to the power failure event, and (v)maintain a transaction log in a random location.
 20. The computer asrecited in claim 19, wherein the data source is one of the followingdevices: fixed media, storage cards, network devices, networks, networkstorage and hard disks.
 21. The computer as recited in claim 19, whereinthe data source is a physical and/or virtual object that is configuredto read and write data.
 22. The computer as recited in claim 19, whereinone of the plurality of interchangeable and/or customizable logic layerscomprises an entry logic layer comprising a set of Application ProtocolInterfaces (APIs) accessible by the application.
 23. The computer asrecited in claim 19, wherein one of the plurality of interchangeableand/or customizable logic layers comprises: a media interface layer,configured to interface directly with the data source.
 24. In a computerhaving an application that issues data requests to a data source, amethod for configuring a file system, the method comprising: augmentingoperational behavior of a file system by substituting and/or modifyingone or more of the plurality of interchangeable and/or customizablelogic layers comprising the file system, respectively, thereby enablingthe file system to function with many different types of data sources;performing logical to physical transformations; detecting and managingpower failure events in a media abstraction layer, wherein the mediaabstraction layer comprises one of the one or more of the plurality ofinterchangeable and/or customizable logic layers and further whereindata integrity and related meta-information are maintained for a lastphysical sector to receive data immediately prior to each of the powerfailure events; and maintaining a transaction log in a random locationfor maintaining data integrity in the event of detecting a power failureevent.
 25. The method as recited in claim 24, wherein the data source isone of the following devices: fixed media, storage cards, networkdevices, networks, network storage and hard disks.
 26. The method asrecited in claim 24, wherein the data source is a physical and/orvirtual object that is configured to read and write data.
 27. The methodas recited in claim 24, wherein the data requests include reading datafrom the data source and/or writing data to the data source.